:facepalm:
This site (and pretty much 90% of other sites) runs off an SQL Database. Information is stored on individual tables in that database. It's then accessed by running an SQL query, which, in the case of one of my sites, looks like this:
PHP:
$result2 = mysql_query ("SELECT * FROM dcomments WHERE ForPost='$postcomment' ORDER BY ID");
Accessing one of my small tables with only 12 entries, the server took only 0.0010 second. Even with a much larger table, it won't be much longer.
So to answer your question, no, all those inactive members won't slow down the site.