Please login or register.

Login with username, password and session length
Advanced search  

News:

Latest sNews - sNews 1.7 - with its own forums - for discussion and user mods.

Pages: [1] 2

Author Topic: [PACKAGE] sNews XE (eXtreme Edition):The fastest, most scalable edition of sNews  (Read 8733 times)

invarbrass

  • Full Member
  • ***
  • Karma: 18
  • Posts: 117
    • http://snews.extremebittorrent.com

Hi,
Everyone knows I did some work on improving sNews' scalability, and speeding it up significantly for a while. Unfortunately, time is a luxury. To cut a long story short: of late I'm finding it very difficult to dedicate time to sNews.
You can grab the latest version of my code (which is almost a month old, as of this writing) from here:
http://snews.extremebittorrent.com/

Here's a brief list of some of the enhancements I've done:
  • Support for MySQL, SQLite 2 and SQLite 3
  • 2-tier caching subsystem to significantly improve site response time
  • Numerous code optimizations to boost speed of execution

There are a lot of files in this distribution. You'll need to put the contents of the upload folder on your website. This folder contains the whole snews distribution + my mods. The DB-schema folder contains the new SQL database schema, for both SQLite and MySQL.
There are a lot of files, here's a brief explanation:
cache and supercache folder: used if you turn on caching. make sure these are world writable.
sn_icons: I pilfered this folder from snews MEMU  Grin
cache_config.php: uncomment the appropriate lines in this file to turn on/off caching.
smartcache_core, smartcache_content: this is the "smarcache engine" for partial caching of page sections. this layer it is deeply embedded within snews.php. speeds up the site considerably.
supercache_begin, supercache_end: this caching layer wraps around the entire snews site, no code change necessary. only index.php is slightly modified. the first time a page is requested, we grab the output buffer, compress the HTML content by packing and cache it. next time the content is requested we simply return the the cached content, no need for snews.php.   ;) In fact you could run the entire site if your database server died and the snews.php was deleted!
db_mysql, db_sqlite, db_sqlite3: as the name implies, abstraction layer for different database back-ends. sqlite3 version uses PDO, which was a royal pain in the a** to adapt to the way snews uses the database.
sql_trace: use it only if you're debugging. delete from your public servers.
html_pack: HTML compressor used by supercache_end.php. it will remove any useless whitespaces, comment to lose fats from sNews' HTML output. but it is also smart; it leaves alone javascripts & textarea-s as it is, since altering them could lead to errors.
google_sitemap, google_sitemap_ping: optimized XML google sitemap generator and pinger. can also generate GZipped sitemap.
ping.php: blog-style ping/trackback. incomplete and unstable. DO NOT USE. or better, fix it and make it work. you'll need the XML-RPC package for this.
config.php: configuration data was extracted from snews.php and put into this file for centralized access.
« Last Edit: March 10, 2008, 03:18:22 PM by invarbrass »
Logged

Keyrocks

  • Doug
  • ULTIMATE member
  • ******
  • Karma: 449
  • Posts: 6019
  • Semantically Challenged
    • snews.ca

Thanks for posting your project work. It serves as a useful repository so that others may improve on it and make use of it.
Your performance contributions to the sNews project have been well read and also spurred a new focus on performance improvements for the upcoming 1.7 Release Candidate.
Don't be a stranger... you are always a welcome sNoozer.  ;D
Logged
Do it now... later may not come.
-------------------------------------------------------------------------------------------------
sNews 1.6 MESU | sNews 1.6 MEMU

invarbrass

  • Full Member
  • ***
  • Karma: 18
  • Posts: 117
    • http://snews.extremebittorrent.com
sNews XE - Database improvements
« Reply #2 on: March 09, 2008, 03:24:38 PM »

I've tested this edition on a large site created with stress-snews. The site contains 20-30 categories, 2000 articles (each article is 80 lines long). All data is stored on MySQL (ISAM) tables. The server is a virtual machine containing Ubuntu 6 LAMP stack with 256 MB RAM.

One of the improvements incorporated into my code is the ability to save/cache database queries. This leads to some significant speed boost.
Here are some benchmark results from a test run of 1000 page-views:
VersionTotal number of queries
Plain vanilla sNews 1.6213020
sNews XE, caching disabled80784
sNews XE, smartcache enabled17016
sNews XE, both smartcache and supercache enabled (first visit)17016
sNews XE, full caching, subsequent visits0

As you can see, even without any form of caching, my code has been able to reduce the total number of queries from 213,000 to 80,000! And if you enable caching, the performance improvement becomes even more significant!

Here's the number of queries executed in a single page-view:
VersionIndex pageArticle page
Plain vanilla sNews 1.6646412
sNews XE, caching disabled7575
sNews XE, smartcache enabled3027
sNews XE, both smartcache and supercache enabled (first visit)3027
sNews XE, full caching, subsequent visits00

The official sNews 1.6 code doesn't scale well on a large site with lots of visitors. The above table gives you a hint at the core problem: sNews executes 400-600 SQL queries on a single page view! In fact, I tried to repeat the benchmarks with higher concurrenct client connections (200-500). As expected, the default sNews-powered site never finished the test. I waited almost an hour before forcefully rebooting the virtual machine. Interestingly, sNews managed to crash the MySQL server and cause database corruption!  :-X

To my surprise, the SQLite version also is not suitable for busy sites. Although it didn't crash the system, the test took too long before I pulled the plug. SQLite is not really recommended if you're planning on a large & busy site.

For high visitor sites, MySQL gives the maximum scalability. (And sNews-XE/MySQL is the only version which was able to complete the stress test  :) )

Additionally, in this release I have optimized and rewrote almost all the SQL queries generated by sNews. For example, the following pseudo-code:
Code: [Select]
SELECT * FROM articles;is replaced by the more optimized (and appropriate) code:
Code: [Select]
SELECT id FROM articles;or this one here:
Code: [Select]
SELECT COUNT(*) FROM ...
Indexes were also added in this release to further improve database performance. Please use the appropriate DDL script in the db-schema folder to create new database. All the indexes were carefully tested using the EXPLAIN statement.

PS: I didn't alter the schema of the comments table  :( (my project didn't have comments enabled). See if you can introduce indices on this table to further optimize the database.  ;)
« Last Edit: March 09, 2008, 03:37:27 PM by invarbrass »
Logged

Joost

  • Guest

Thanks Invarbrass,

Like Keyrocks already noted, a great deal of of you work will be in the next release. So thanks.
Seems we have the same feelings about 'comments'. Not only about not wanting them  ;D, but also about the table structure.

I wish you all the luck with study/research and keep in touch.
Logged

invarbrass

  • Full Member
  • ***
  • Karma: 18
  • Posts: 117
    • http://snews.extremebittorrent.com

thanks keys and joost for the encouragement! this people behind snews are very generous and helpful  ;D
Logged

jlhaslip

  • Sr. Member
  • ****
  • Karma: 16
  • Posts: 374
    • My snews with AEF Forum site

Your efforts towards enhancing snews is appreciated.

Best wishes in your endeavors, invarbrass. Thank you.
Logged
Yes! I have no siggy.

BuLe Bali

  • Full Member
  • ***
  • Karma: 2
  • Posts: 100
    • A World Full of Doubts

Thanks invarbrass. You are awesome.
Logged
This world are full of doubts! No systems is secure. No systems!
@yohanesbule

Sven

  • ULTIMATE member
  • ******
  • Karma: 88
  • Posts: 2029
  • Chasing MY bugs!
    • hiseo.fr - rédacteur Web

Is this version still up to date Dr invarbrass? :)

funlw65

  • Hero Member
  • *****
  • Karma: 96
  • Posts: 771
    • Country Lab

Maybe I'm wrong but I think he dropped the SQLite support...
Logged

Joost

  • Guest

Maybe I'm wrong but I think he dropped the SQLite support...

True, Invarbrass built this package primarily for personal use. It is just one of his contributions to the community.
Logged

Sven

  • ULTIMATE member
  • ******
  • Karma: 88
  • Posts: 2029
  • Chasing MY bugs!
    • hiseo.fr - rédacteur Web

Howdee Dudes?
Our sNews doctor should comme overhere more often. :)
Armen has recenttly talked about a Vanilla version : I think it's this one.
I'm gonna try it for a "Carnet" (a micro-blog) in a subdomain of Hiseo to talk about my hobbies.
Hope it's gonna fit.
Have a nice day pals. 8)

Armen

  • Sr. Member
  • ****
  • Karma: 41
  • Posts: 338
    • http://www.funnydays.ru

Howdee Dudes?
Our sNews doctor should comme overhere more often. :)
Armen has recenttly talked about a Vanilla version : I think it's this one.
I'm gonna try it for a "Carnet" (a micro-blog) in a subdomain of Hiseo to talk about my hobbies.
Hope it's gonna fit.
Have a nice day pals. 8)

Vanilla means "Default". The one by Luka in downloads section.
Logged
Now ogres, oh, they're much worse. They'll make a suit from your freshly peeled skin. They'll shave your liver, squeeze the jelly from your eyes... Actually, it's quite good on toast.

Keyrocks

  • Doug
  • ULTIMATE member
  • ******
  • Karma: 449
  • Posts: 6019
  • Semantically Challenged
    • snews.ca

Vanilla means "Default". The one by Luka in downloads section.

I believe "Vanilla" refers to the default template... not necessarily the complete file-set... nothing more.  :)
Logged
Do it now... later may not come.
-------------------------------------------------------------------------------------------------
sNews 1.6 MESU | sNews 1.6 MEMU

Sven

  • ULTIMATE member
  • ******
  • Karma: 88
  • Posts: 2029
  • Chasing MY bugs!
    • hiseo.fr - rédacteur Web

Hi my dear ogre,
hi Doug.
And thanks to both of both for the clarification. :-\

alMubarmij

  • Newbie
  • *
  • Karma: 0
  • Posts: 13
Great work..
« Reply #14 on: October 15, 2008, 07:45:07 PM »

Great work..
But I think there are some bugs:

1- There is no CSS file.
2- I got 404 error with most links.
3- No installation form.
4- No design template
5- No sample SQLite database.
Logged
Pages: [1] 2