Github/stackdump
1
0
Fork 0
mirror of https://github.com/djohnlewis/stackdump synced 2024-12-04 23:17:37 +00:00
Code Issues Projects Releases Wiki Activity
166 Commits 5 Branches 5 Tags 92 MiB
01f9b10c27
Commit Graph

166 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Samuel Lai
01f9b10c27 Fixed #7. Turns out post IDs are not unique across sites. 
This change will require re-indexing of all sites unfortunately. On the upside, more questions to browse!
 2014-02-27 17:57:34 +11:00
Sam
cdb93e6f68 Merged changes. 2014-02-16 01:04:19 +11:00
Sam
0990e00852 Added an original copy of pysolr.py so the custom changes can be worked out. 2014-02-16 01:03:05 +11:00
Samuel Lai
92e359174a Added some notes on importing StackOverflow on Windows. 2013-12-12 17:29:55 +11:00
Samuel Lai
c521fc1627 Added tag v1.2 for changeset 240affa260a1 2013-11-30 18:06:37 +11:00
Sam
722d4125e7 Added section in README re new PowerShell scripts. 
Also fixed formatting and wording.
 2013-12-01 03:43:58 +11:00
Sam
ce3eb04270 Updated README with v1.2 changes and SO import stats. 2013-12-01 03:33:40 +11:00
Samuel Lai
9613caa8d1 Changed settings so Solr now only listens on localhost, not all interfaces. 2013-11-29 15:18:55 +11:00
Samuel Lai
2583afeb90 Removed more redundant date/time parsing. 2013-11-29 15:11:32 +11:00
Samuel Lai
522e1ff4f2 Fixed bug in script where the directory change was not reverted when script exited. 2013-11-29 15:06:10 +11:00
Samuel Lai
36eb8d3980 Changed the name of the stackdump schema to something better than 'Example'. 2013-11-29 15:05:31 +11:00
Samuel Lai
a597b2e588 Merge import-perf-improvements branch to default. 2013-11-29 13:01:41 +11:00
Samuel Lai
4a9c4504b3 Updated bad docs. 2013-11-29 12:57:06 +11:00
Samuel Lai
77dd2def42 Oops, forgot to re-instate the comment index during the backout. 2013-11-29 01:42:17 +11:00
Samuel Lai
75a216f5a4 Backed out the comments-batching change. 
It was causing weird perf issues and errors. Didn't really seem like it made things faster; if anything, things became slower.
 2013-11-29 01:12:09 +11:00
Samuel Lai
bf09e36928 Changed other models to avoid unnecessary date/time parsing. 
Added PRAGMA statements for comments table and changed flow so the siteId_postId index is now created after data has been inserted.
 2013-11-29 00:18:54 +11:00
Samuel Lai
cdb8d96508 Comments are now committed in batches and using a 'prepared' statement via executemany. 
Also fixed a Windows compatibility bug with the new temp comments db and a bug with the webapp now that the Comment model has moved. Dates are also no longer parsed from their ISO form for comments; instead left as strings and parsed by SQLObject internally as needed.
 2013-11-28 23:51:53 +11:00
Samuel Lai
5868c8e328 Fixed settings for Windows compatibility. 2013-11-28 22:06:33 +11:00
Samuel Lai
8e3d21f817 Fixed settings for Windows compatibility. 2013-11-28 22:06:33 +11:00
Samuel Lai
2fea457b06 Added PowerShell equivalents to launch and manage Stackdump on Windows. 2013-11-28 21:53:45 +11:00
Samuel Lai
6469691e4b Added PowerShell equivalents to launch and manage Stackdump on Windows. 2013-11-28 21:53:45 +11:00
Samuel Lai
65394ac516 More minor fixes. Really should get Stackdump set-up on my dev machine. 2013-11-28 15:07:05 +11:00
Samuel Lai
bcf1d7c71a Again. Forgot to fix site->siteId rename. 2013-11-28 14:39:25 +11:00
Samuel Lai
d36146ae46 More bugs - forgot to rename uses when renaming Comment.site to siteId 2013-11-28 14:38:21 +11:00
Samuel Lai
e1272ce58a Oops, bug with closing temp_db file handle. 2013-11-28 14:35:24 +11:00
Samuel Lai
bff7e13d83 Comment data used during importing is now stored in a separate database to make it easier to delete them afterwards. 2013-11-28 14:23:55 +11:00
Samuel Lai
c0766de8d4 Skips valid XML character scrubbing if configured for faster performance. 2013-11-28 14:01:00 +11:00
Samuel Lai
644269dd5d Added PyCharm project files to the ignore list. 2013-11-28 13:54:47 +11:00
Sam
6bbf0d7b28 Removed a big duplicate file in Solr. 2013-10-22 23:36:46 +11:00
Sam
71c875437e Added tag v1.1 for changeset 3ad1ff15b528 2013-10-22 23:21:20 +11:00
Sam
e78e70e5ac Updated README for v1.1. 2013-10-22 23:20:45 +11:00
Sam
77c76ea9d4 Grrr, forgot to add a file to the previous changeset. 
This adds the template that is used when bad Solr syntax is encountered.
 2013-10-22 23:20:23 +11:00
Sam
7dc7b7b5bd Solr syntax errors are now handled nicely. 
Fixes #4.
 2013-10-22 22:43:57 +11:00
Sam
645b24f370 Question permalinks are now recognised and internally linked. 
Fixes #3.
 2013-10-22 22:03:09 +11:00
Sam
f067353f62 Added answer permalinks and ability to rewrite internal answer permalinks. 
This includes a new field in the Solr schema, so a re-index is required after this changeset.

Fixes #1
 2013-10-22 21:59:49 +11:00
Sam
4e924f6bd8 Removed the extracted solr-webapp files from the repository. 
The files are automatically extracted on launch from the war file.
 2013-10-22 20:56:42 +11:00
Sam
bc5642af71 Removed the Solr log file. 
Luckily there was nothing sensitive in there... I don't think.
 2013-10-22 20:49:19 +11:00
Sam
09edf16128 Added missing rule to render external links in comments differently. 2013-10-22 08:45:07 +11:00
Sam
436b86b234 Upgrade Apache Solr to 4.5.0 and PySolr to 3.1.0. 
All Solr indexes will need to be re-created.
 2013-10-22 08:44:37 +11:00
Sam
e88e6a31a0 Added a comment about configuring SELinux to allow JRE 7 to run. 2013-10-14 07:44:10 +11:00
Sam
706fd5ef9d Fixed a bug where site names with non-ASCII characters caused a crash. 2013-10-14 07:32:45 +11:00
Sam
9cac41929b Added quotes in a bunch of places so things work with paths containing spaces. 2013-10-14 07:31:50 +11:00
Sam
3615a72310 Removed the -server arg for starting Solr. 
This causes issues with JVMs that don't have the 'server' VM built-in,
e.g. the Windows i386 one. The JVM automatically selects the server
VM anyway when the host characteristics are enough anyway.
 2013-10-14 07:24:10 +11:00
Sam
a472517736 Refactored the import_site command; now caters for filenames of different case (the case changed in 2013 dumps). 2013-09-24 18:07:55 +10:00
Samuel Lai
70fa72b04e Added new external components to README. 2012-12-15 22:53:21 +11:00
Samuel Lai
b667ea0165 Added Mathjax support for math.stackexchange.com. 
Equations and expressions are only rendered in question view at the moment.
 2012-12-15 22:47:46 +11:00
Samuel Lai
993bee4fc1 Added markdown parsing for comments so links in comments now appear properly. 
Also rewrote part of the HTML rewriting code so it doesn't introduce an additional wrapping element in the output which was added due to a html5lib requirements on input.
 2012-12-15 21:43:06 +11:00
Samuel Lai
5ac8492f38 Improved the README file with details on disk space requirements and configuration. 2012-08-25 17:05:13 +10:00
Samuel Lai
36a605711e Added StackExchange question and user URLs to pages as a tooltip to comply with attribution requirements. 
Attribution requirements actually state that hyperlinked URLs should be used, but they would be rather useless in this app so this is an alternative.
 2012-08-25 16:57:01 +10:00
Samuel Lai
e4b2ee80a0 Fixed a bug where extra html/head/body tags were added for every question and answer due to the HTML re-writing process. 2012-08-25 16:37:03 +10:00