Github/stackdump
1
0
Fork 0
mirror of https://github.com/djohnlewis/stackdump synced 2025-01-22 14:41:39 +00:00
Code Issues Projects Releases Wiki Activity
160 Commits 5 Branches 5 Tags 92 MiB
comments-in-solr
Commit Graph

119 Commits

Author SHA1 Message Date
Samuel Lai
665acbb4eb Removed more redundant date/time parsing. 2013-11-29 15:11:32 +11:00
Samuel Lai
3748d24f42 Modified import_site to store comments in a Solr core instead of SQLite DB. 
manage_site yet to be updated. Seems to be a bit slower than SQLite though. Not sure of performance with larger datasets.
 2013-11-29 14:59:18 +11:00
Samuel Lai
4a9c4504b3 Updated bad docs. 2013-11-29 12:57:06 +11:00
Samuel Lai
77dd2def42 Oops, forgot to re-instate the comment index during the backout. 2013-11-29 01:42:17 +11:00
Samuel Lai
75a216f5a4 Backed out the comments-batching change. 
It was causing weird perf issues and errors. Didn't really seem like it made things faster; if anything, things became slower.
 2013-11-29 01:12:09 +11:00
Samuel Lai
bf09e36928 Changed other models to avoid unnecessary date/time parsing. 
Added PRAGMA statements for comments table and changed flow so the siteId_postId index is now created after data has been inserted.
 2013-11-29 00:18:54 +11:00
Samuel Lai
cdb8d96508 Comments are now committed in batches and using a 'prepared' statement via executemany. 
Also fixed a Windows compatibility bug with the new temp comments db and a bug with the webapp now that the Comment model has moved. Dates are also no longer parsed from their ISO form for comments; instead left as strings and parsed by SQLObject internally as needed.
 2013-11-28 23:51:53 +11:00
Samuel Lai
8e3d21f817 Fixed settings for Windows compatibility. 2013-11-28 22:06:33 +11:00
Samuel Lai
65394ac516 More minor fixes. Really should get Stackdump set-up on my dev machine. 2013-11-28 15:07:05 +11:00
Samuel Lai
bcf1d7c71a Again. Forgot to fix site->siteId rename. 2013-11-28 14:39:25 +11:00
Samuel Lai
d36146ae46 More bugs - forgot to rename uses when renaming Comment.site to siteId 2013-11-28 14:38:21 +11:00
Samuel Lai
e1272ce58a Oops, bug with closing temp_db file handle. 2013-11-28 14:35:24 +11:00
Samuel Lai
bff7e13d83 Comment data used during importing is now stored in a separate database to make it easier to delete them afterwards. 2013-11-28 14:23:55 +11:00
Samuel Lai
c0766de8d4 Skips valid XML character scrubbing if configured for faster performance. 2013-11-28 14:01:00 +11:00
Sam
77c76ea9d4 Grrr, forgot to add a file to the previous changeset. 
This adds the template that is used when bad Solr syntax is encountered.
 2013-10-22 23:20:23 +11:00
Sam
7dc7b7b5bd Solr syntax errors are now handled nicely. 
Fixes #4.
 2013-10-22 22:43:57 +11:00
Sam
645b24f370 Question permalinks are now recognised and internally linked. 
Fixes #3.
 2013-10-22 22:03:09 +11:00
Sam
f067353f62 Added answer permalinks and ability to rewrite internal answer permalinks. 
This includes a new field in the Solr schema, so a re-index is required after this changeset.

Fixes #1
 2013-10-22 21:59:49 +11:00
Sam
09edf16128 Added missing rule to render external links in comments differently. 2013-10-22 08:45:07 +11:00
Sam
436b86b234 Upgrade Apache Solr to 4.5.0 and PySolr to 3.1.0. 
All Solr indexes will need to be re-created.
 2013-10-22 08:44:37 +11:00
Sam
706fd5ef9d Fixed a bug where site names with non-ASCII characters caused a crash. 2013-10-14 07:32:45 +11:00
Sam
a472517736 Refactored the import_site command; now caters for filenames of different case (the case changed in 2013 dumps). 2013-09-24 18:07:55 +10:00
Samuel Lai
b667ea0165 Added Mathjax support for math.stackexchange.com. 
Equations and expressions are only rendered in question view at the moment.
 2012-12-15 22:47:46 +11:00
Samuel Lai
993bee4fc1 Added markdown parsing for comments so links in comments now appear properly. 
Also rewrote part of the HTML rewriting code so it doesn't introduce an additional wrapping element in the output which was added due to a html5lib requirements on input.
 2012-12-15 21:43:06 +11:00
Samuel Lai
36a605711e Added StackExchange question and user URLs to pages as a tooltip to comply with attribution requirements. 
Attribution requirements actually state that hyperlinked URLs should be used, but they would be rather useless in this app so this is an alternative.
 2012-08-25 16:57:01 +10:00
Samuel Lai
e4b2ee80a0 Fixed a bug where extra html/head/body tags were added for every question and answer due to the HTML re-writing process. 2012-08-25 16:37:03 +10:00
Samuel Lai
af28d3e403 Added in a setting to control the number of random questions to show on the search pages. 2012-08-25 16:20:57 +10:00
Samuel Lai
c240356a7b Added a check and a nicer message for when the management commands can't connect to solr. 2012-08-24 18:48:17 +10:00
Samuel Lai
049e857159 Handled another exception that may occur if no data has been imported. 2012-08-19 12:47:42 +10:00
Samuel Lai
16e5530a82 Modified download_site_info script to create the data directory if it doesn't exist. 2012-08-19 12:30:33 +10:00
Samuel Lai
1b27784a8c Added an error page for when Stackdump fails to connect to Solr. 
Also unified the error pages and added a generic 500 error page.
 2012-08-19 00:09:35 +10:00
Samuel Lai
e0c96a5c5f Fixed a minor styling issue with question titles on search result pages. 2012-08-18 23:38:15 +10:00
Samuel Lai
01b0dcae39 Fixed a minor CSS spacing issue between the 'show more comments' block and the moderation message. 2012-08-18 21:40:29 +10:00
Samuel Lai
c1a5382622 Modified app to use a settings file. 
This allows users to change the Solr URL and other things in one spot.
 2012-08-18 21:39:17 +10:00
Samuel Lai
46100e7f01 Fixed a small bug where the 'serving media from' message was printed twice. 2012-08-18 21:08:28 +10:00
Samuel Lai
4430997467 Amended import instructions to account for the command changes in previous commit. 2012-08-18 21:00:58 +10:00
Samuel Lai
e8adaa9b54 Renamed the commands directory and added a script to make them easier to call. 
Also deleted the get_sites script as it wasn't very useful, and renamed others
to be more meaningful.
 2012-08-18 20:50:13 +10:00
Samuel Lai
e776e95d84 Added an alias for questions for StackExchange style URLs. 
This means site_key/questions/question_id is redirected to site_key/question_id.
 2012-08-18 20:23:14 +10:00
Samuel Lai
5fc56e4329 Added check to ensure the accepted answer to a question actually exists. 
There may be times when it does not exist, e.g. when a question has been merged.
 2012-08-18 20:18:26 +10:00
Samuel Lai
9b9b71077c Added informative message when Stackdump is disabled during a site import. 2012-08-18 20:17:15 +10:00
Samuel Lai
2954dd47ba Added a message for questions that have been closed. 2012-08-18 18:40:18 +10:00
Samuel Lai
6181d83cf3 Added a confirmation prompt when importing so the user can confirm site details. 2012-08-18 18:17:00 +10:00
Samuel Lai
9bcac3f92a Fixed some grammar errors in the footer text. 2012-08-18 17:48:55 +10:00
Samuel Lai
ad5f11260a Changed wording from 'posts' to 'questions' on search results pages. 2012-08-18 17:44:57 +10:00
Samuel Lai
827445105b Excess comments (defaults to any over 3) are now hidden by default. 
They can be shown by clicking on the 'show comments' link.
 2012-08-18 17:44:01 +10:00
Samuel Lai
3d515f51b1 For results with lots of pages, only a limited set of page numbers are rendered. 2012-08-12 16:32:42 +10:00
Samuel Lai
3944261eef Fixed a bug where uncommitted entries from a previously failed import were committed in a later, successful import. 2012-08-12 16:31:30 +10:00
Samuel Lai
1f29fd9113 Modified import.py so it no longer relies on readme.txt. 
readme.txt files were dropped as of the August 2012 data dump.
 2012-08-12 15:40:48 +10:00
Samuel Lai
dd24d98b39 Upgraded Bottle.py to 0.10.11 and CherryPy to 3.2.2. 2012-08-12 14:57:25 +10:00
Samuel Lai
26b803e119 Improved import speed by ~9-fold by actually committing every 1000 questions. 
There was an error made where although questions were only checked for completion every 1000 rows, each completed question was committed separately, resulting in far too many solr calls.

Also modified process to only commit entries in solr at the end, after the database transaction is committed. This means if the process is aborted mid-way through, there won't be orphaned data in solr any more.
 2012-08-12 14:13:15 +10:00