sNews Forum

sNews 1.6 (previous version) => Mods/addons => Topic started by: invarbrass on March 09, 2008, 03:00:14 PM

Title: [PACKAGE] sNews XE (eXtreme Edition):The fastest, most scalable edition of sNews
Post by: invarbrass on March 09, 2008, 03:00:14 PM
Hi,
Everyone knows I did some work on improving sNews' scalability, and speeding it up significantly for a while. Unfortunately, time is a luxury. To cut a long story short: of late I'm finding it very difficult to dedicate time to sNews.
You can grab the latest version of my code (which is almost a month old, as of this writing) from here:
http://snews.extremebittorrent.com/ (http://snews.extremebittorrent.com/)

Here's a brief list of some of the enhancements I've done:

There are a lot of files in this distribution. You'll need to put the contents of the upload folder on your website. This folder contains the whole snews distribution + my mods. The DB-schema folder contains the new SQL database schema, for both SQLite and MySQL.
There are a lot of files, here's a brief explanation:
cache and supercache folder: used if you turn on caching. make sure these are world writable.
sn_icons: I pilfered this folder from snews MEMU  Grin
cache_config.php: uncomment the appropriate lines in this file to turn on/off caching.
smartcache_core, smartcache_content: this is the "smarcache engine" for partial caching of page sections. this layer it is deeply embedded within snews.php. speeds up the site considerably.
supercache_begin, supercache_end: this caching layer wraps around the entire snews site, no code change necessary. only index.php is slightly modified. the first time a page is requested, we grab the output buffer, compress the HTML content by packing and cache it. next time the content is requested we simply return the the cached content, no need for snews.php.   ;) In fact you could run the entire site if your database server died and the snews.php was deleted!
db_mysql, db_sqlite, db_sqlite3: as the name implies, abstraction layer for different database back-ends. sqlite3 version uses PDO, which was a royal pain in the a** to adapt to the way snews uses the database.
sql_trace: use it only if you're debugging. delete from your public servers.
html_pack: HTML compressor used by supercache_end.php. it will remove any useless whitespaces, comment to lose fats from sNews' HTML output. but it is also smart; it leaves alone javascripts & textarea-s as it is, since altering them could lead to errors.
google_sitemap, google_sitemap_ping: optimized XML google sitemap generator and pinger. can also generate GZipped sitemap.
ping.php: blog-style ping/trackback. incomplete and unstable. DO NOT USE. or better, fix it and make it work. you'll need the XML-RPC package for this.
config.php: configuration data was extracted from snews.php and put into this file for centralized access.
Title: Re: sNews XE (eXtreme Edition) - The fastest, most scalable edition of sNews!
Post by: Keyrocks on March 09, 2008, 03:15:02 PM
Thanks for posting your project work. It serves as a useful repository so that others may improve on it and make use of it.
Your performance contributions to the sNews project have been well read and also spurred a new focus on performance improvements for the upcoming 1.7 Release Candidate.
Don't be a stranger... you are always a welcome sNoozer.  ;D
Title: sNews XE - Database improvements
Post by: invarbrass on March 09, 2008, 03:24:38 PM
I've tested this edition on a large site created with stress-snews. The site contains 20-30 categories, 2000 articles (each article is 80 lines long). All data is stored on MySQL (ISAM) tables. The server is a virtual machine containing Ubuntu 6 LAMP stack with 256 MB RAM.

One of the improvements incorporated into my code is the ability to save/cache database queries. This leads to some significant speed boost.
Here are some benchmark results from a test run of 1000 page-views:
VersionTotal number of queries
Plain vanilla sNews 1.6213020
sNews XE, caching disabled80784
sNews XE, smartcache enabled17016
sNews XE, both smartcache and supercache enabled (first visit)17016
sNews XE, full caching, subsequent visits0

As you can see, even without any form of caching, my code has been able to reduce the total number of queries from 213,000 to 80,000! And if you enable caching, the performance improvement becomes even more significant!

Here's the number of queries executed in a single page-view:
VersionIndex pageArticle page
Plain vanilla sNews 1.6646412
sNews XE, caching disabled7575
sNews XE, smartcache enabled3027
sNews XE, both smartcache and supercache enabled (first visit)3027
sNews XE, full caching, subsequent visits00

The official sNews 1.6 code doesn't scale well on a large site with lots of visitors. The above table gives you a hint at the core problem: sNews executes 400-600 SQL queries on a single page view! In fact, I tried to repeat the benchmarks with higher concurrenct client connections (200-500). As expected, the default sNews-powered site never finished the test. I waited almost an hour before forcefully rebooting the virtual machine. Interestingly, sNews managed to crash the MySQL server and cause database corruption!  :-X

To my surprise, the SQLite version also is not suitable for busy sites. Although it didn't crash the system, the test took too long before I pulled the plug. SQLite is not really recommended if you're planning on a large & busy site.

For high visitor sites, MySQL gives the maximum scalability. (And sNews-XE/MySQL is the only version which was able to complete the stress test  :) )

Additionally, in this release I have optimized and rewrote almost all the SQL queries generated by sNews. For example, the following pseudo-code:
Code: [Select]
SELECT * FROM articles;is replaced by the more optimized (and appropriate) code:
Code: [Select]
SELECT id FROM articles;or this one here:
Code: [Select]
SELECT COUNT(*) FROM ...
Indexes were also added in this release to further improve database performance. Please use the appropriate DDL script in the db-schema folder to create new database. All the indexes were carefully tested using the EXPLAIN statement.

PS: I didn't alter the schema of the comments table  :( (my project didn't have comments enabled). See if you can introduce indices on this table to further optimize the database.  ;)
Title: Re: sNews XE (eXtreme Edition) - The fastest, most scalable edition of sNews!
Post by: Joost on March 09, 2008, 07:21:42 PM
Thanks Invarbrass,

Like Keyrocks already noted, a great deal of of you work will be in the next release. So thanks.
Seems we have the same feelings about 'comments'. Not only about not wanting them  ;D, but also about the table structure.

I wish you all the luck with study/research and keep in touch.
Title: Re: sNews XE (eXtreme Edition) - The fastest, most scalable edition of sNews!
Post by: invarbrass on March 10, 2008, 01:55:22 PM
thanks keys and joost for the encouragement! this people behind snews are very generous and helpful  ;D
Title: Re: [PACKAGE] sNews XE (eXtreme Edition):The fastest, most scalable edition of sNews
Post by: jlhaslip on March 10, 2008, 08:36:27 PM
Your efforts towards enhancing snews is appreciated.

Best wishes in your endeavors, invarbrass. Thank you.
Title: Re: [PACKAGE] sNews XE (eXtreme Edition):The fastest, most scalable edition of sNews
Post by: BuLe Bali on July 11, 2008, 09:20:02 AM
Thanks invarbrass. You are awesome.
Title: Re: [PACKAGE] sNews XE (eXtreme Edition):The fastest, most scalable edition of sNews
Post by: Sven on September 21, 2008, 06:17:44 PM
Is this version still up to date Dr invarbrass? :)
Title: Re: [PACKAGE] sNews XE (eXtreme Edition):The fastest, most scalable edition of sNews
Post by: funlw65 on September 21, 2008, 06:21:27 PM
Maybe I'm wrong but I think he dropped the SQLite support...
Title: Re: [PACKAGE] sNews XE (eXtreme Edition):The fastest, most scalable edition of s
Post by: Joost on September 21, 2008, 06:33:19 PM
Maybe I'm wrong but I think he dropped the SQLite support...

True, Invarbrass built this package primarily for personal use. It is just one of his contributions to the community.
Title: Re: [PACKAGE] sNews XE (eXtreme Edition):The fastest, most scalable edition of sNews
Post by: Sven on September 22, 2008, 07:52:08 AM
Howdee Dudes?
Our sNews doctor should comme overhere more often. :)
Armen has recenttly talked about a Vanilla version : I think it's this one.
I'm gonna try it for a "Carnet" (a micro-blog) in a subdomain of Hiseo to talk about my hobbies.
Hope it's gonna fit.
Have a nice day pals. 8)
Title: Re: [PACKAGE] sNews XE (eXtreme Edition):The fastest, most scalable edition of sNews
Post by: Armen on September 22, 2008, 12:25:09 PM
Howdee Dudes?
Our sNews doctor should comme overhere more often. :)
Armen has recenttly talked about a Vanilla version : I think it's this one.
I'm gonna try it for a "Carnet" (a micro-blog) in a subdomain of Hiseo to talk about my hobbies.
Hope it's gonna fit.
Have a nice day pals. 8)

Vanilla means "Default". The one by Luka in downloads section.
Title: Re: [PACKAGE] sNews XE (eXtreme Edition):The fastest, most scalable edition of sNews
Post by: Keyrocks on September 22, 2008, 02:35:19 PM
Vanilla means "Default". The one by Luka in downloads section.

I believe "Vanilla" refers to the default template... not necessarily the complete file-set... nothing more.  :)
Title: Re: [PACKAGE] sNews XE (eXtreme Edition):The fastest, most scalable edition of sNews
Post by: Sven on September 22, 2008, 03:27:26 PM
Hi my dear ogre,
hi Doug.
And thanks to both of both for the clarification. :-\
Title: Great work..
Post by: alMubarmij on October 15, 2008, 07:45:07 PM
Great work..
But I think there are some bugs:

1- There is no CSS file.
2- I got 404 error with most links.
3- No installation form.
4- No design template
5- No sample SQLite database.
Title: Re: Great work..
Post by: Keyrocks on October 15, 2008, 09:17:07 PM
Great work..
But I think there are some bugs:

1- There is no CSS file.
2- I got 404 error with most links.
3- No installation form.
4- No design template
5- No sample SQLite database.

Please give us a complete list of all the files, by name, that came in the ZIP package you have.
Title: Re: [PACKAGE] sNews XE (eXtreme Edition):The fastest, most scalable edition of s
Post by: alMubarmij on October 15, 2008, 09:44:07 PM
These files:
http://snews.extremebittorrent.com/snews-xe-feb-11-2008.rar
Title: Re: [PACKAGE] sNews XE (eXtreme Edition):The fastest, most scalable edition of sNews
Post by: centered on July 09, 2009, 06:46:27 PM
This thread and package is pretty old but the code and techniques are still usable today. I have some code improvements below.  This code allows the SQLite DB version to be automatically switchable and adds some more functionality.

Code: [Select]
<?php

define
'__sqlite3'in_array'pdo_sqlite'get_loaded_extensions() ) );
define'__sqlite2'in_array'SQLite'get_loaded_extensions() ) );

$dbHandler FALSE;
$dbName $db['dbname']; // from configuration file or db array
$sqlFile 'sql file';
__dataPath 'path/to/your/db/folder';

// For SQLite 3
if ( __sqlite3 && !file_exists(__dataPath$dbName '_2.db') ) { 
function dbOpen() {
global $dbName$dbHandler;
$dbHandler = new PDO'sqlite:' __dataPath$dbName '_3.db' ) or die( dbError() );
return $dbHandler;
}

function dbError() {
global $dbHandler;
$error $dbHandler->errorInfo();
return $error['2'];
}

function dbQuery$query ) {
global $dbHandler;  
return $dbHandler->query$query );
}

function dbArrayQuery$query ) {
global $dbHandler;  
$result $dbHandler->query$query ); 
$result $result->fetchAll(PDO::FETCH_ASSOC);
return $result;
}

function dbFetchArray$query_result ) { 
return $query_result->fetch();
}

function dbFetchAssoc$query_result ) { 
return $query_result->fetch(PDO::FETCH_ASSOC);
}

function dbEscape$string ) {
if( function_exists'sqlite_escape_string' ) ) {
$string sqlite_escape_string$string );
} else {
$string str_replace"'""''"$string );
}
return $string;
}
} elseif ( 
__sqlite2 ) { 
function dbOpen() {
global $dbName$dbHandler;
$dbHandler sqlite_open__dataPath $dbName '_2.db'0666) or die( dbError() );
return $dbHandler;
}

function dbError() {
global $dbHandler;
return sqlite_error_stringsqlite_last_error$dbHandler ) );
}

function dbQuery$query ) {
global $dbHandler;  
return sqlite_query$dbHandler$query );
}

function dbArrayQuery$query ) {
global $dbHandler;    
return sqlite_array_query$dbHandler$querySQLITE_ASSOC ); 
}

function dbFetchArray$query_result ) { 
return sqlite_fetch_array$query_resultSQLITE_BOTH );
}

function dbFetchAssoc$query_result ) { 
return sqlite_fetch_array$query_resultSQLITE_ASSOC );
}

function dbEscape$string ) {
return sqlite_escape_string$string );
}
}

?>

Gotcha = sqlite num_row is not here.  PDO version for Sqlite 3 (rowCount) doesn't work too well so I scrapped it completely.  If you require a count, use the query:

select (count(id) from table where column = value) as num_rows....

Gotcha #2 - If you have a sqlite2 database running in a sqlite3 environment, the script would create a new sqlite 3 database.... to use the existing sqlite 2 database, I needed hack some changes like the underscore version # to differentiate the different versions.  If you wanted to know the version  you could use a fucntion like this (granted you have 1 db - note tested for more than 1):
Code: [Select]
function getdbVersion() {
$dir = opendir('data path');
if ($dir) {
while (false !== ($file = readdir($dir))) {
if ( preg_match('|\.db$|', $file)&& !is_dir($path.'/'.$file) ) {
$file = substr("$file",0,strpos($file, '.db'));
$version = substr($file, -1);
}
}
}
return $version;
}
$dbVersion = getdbVersion();
Title: Re: [PACKAGE] sNews XE (eXtreme Edition):The fastest, most scalable edition of sNews
Post by: Cocor on October 21, 2012, 06:03:49 PM
Hi can any one reupload this package? I want to try to use this project files. Thanks!
Title: Re: [PACKAGE] sNews XE (eXtreme Edition):The fastest, most scalable edition of sNews
Post by: Keyrocks on October 22, 2012, 03:20:30 AM
Hi can any one reupload this package? I want to try to use this project files. Thanks!

You may need to try contacting the author directly using the email address on the author's Profile Page (http://snewscms.com/forum/index.php?action=profile;u=8582). The author has not logged in here since October 2008.  8)