Github/stackdump
1
0
Fork 0
mirror of https://github.com/djohnlewis/stackdump synced 2025-04-04 16:53:27 +00:00
Code Issues Projects Releases Wiki Activity

Refactored the 'no data' page so the import instructions are accessible even after you've imported your initial dump(s) via a link in the footer.

Browse Source
This commit is contained in:
Samuel Lai 2012-02-12 14:14:15 +11:00
parent ea0c592fed
commit caebcabd89
6 changed files with 105 additions and 57 deletions

4
python/media/css/main.css

@ -320,6 +320,10 @@ h1.answers {
padding: 7px 10px; padding: 7px 10px;
} }
#footer .footer-right {
float: right;
}
.nodata.row { .nodata.row {
margin-top: 50px; margin-top: 50px;
margin-bottom: 20px; margin-bottom: 20px;

11
python/src/stackdump/app.py

@ -214,6 +214,17 @@ def index():
return render_template('index.html', context) return render_template('index.html', context)
# this method MUST sit above the site_index and other methods below so it
# cannot be subverted by a site with a site key of 'import'.
@get('/import')
@get('/import/')
@uses_templates
def import_data():
'''\
Renders the 'how to import data' page.
'''
return render_template('import_data.html')
@get('/:site_key#[\w\.]+#') @get('/:site_key#[\w\.]+#')
@get('/:site_key#[\w\.]+#/') @get('/:site_key#[\w\.]+#/')
@uses_templates @uses_templates

12
python/src/stackdump/templates/base.html

@ -41,10 +41,16 @@
</div> </div>
<div id="footer"> <div id="footer">
Stackdump is licensed under <p class="footer-right">
<a href="{{ SETTINGS.APP_URL_ROOT }}import">How do I import data?</a>
</p>
<a href="https://bitbucket.org/samuel.lai/stackdump/">Stackdump</a> is
licensed under
<a href="http://en.wikipedia.org/wiki/MIT_License">MIT License</a>. The <a href="http://en.wikipedia.org/wiki/MIT_License">MIT License</a>. The
contents is licensed under <a href="http://creativecommons.org/licenses/by-sa/3.0/"> <a href="http://www.stackexchange.com">StackExchange</a> content is
Creative Commons [cc-wiki]</a>. licensed under
<a href="http://creativecommons.org/licenses/by-sa/3.0/">Creative
Commons [cc-wiki]</a>.
</div> </div>
</body> </body>
</html> </html>

26
python/src/stackdump/templates/import_data.html Normal file

@ -0,0 +1,26 @@
{% extends 'base.html' %}
{% block title %}
Stackdump - Import data instructions
{% endblock %}
{% block body %}
<div class="row nodata">
<div class="span16">
<h1>Importing data into Stackdump</h1>
<p>
Use the procedure below to import new sites into this Stackdump
instance, or to upgrade the content to a later version.
</p>
<p>
If you are upgrading, just follow the steps below. There are no
special steps to perform for upgrades; the process below will
automatically remove the existing site and import the new
version.
</p>
{% include 'includes/import_data.html.inc' %}
</div>
</div>
{% endblock %}

54
python/src/stackdump/templates/includes/import_data.html.inc Normal file

@ -0,0 +1,54 @@
<h2>Get the StackExchange data dump</h2>
<p>
StackExchange data dumps are distributed using BitTorrent. You
will need a BitTorrent client like
<a href="http://www.utorrent.com">uTorrent</a> to download it.
</p>
<p>
The data dumps can be downloaded from
<a href="http://www.clearbits.net/creators/146-stack-exchange-data-dump">http://www.clearbits.net/creators/146-stack-exchange-data-dump</a>.
</p>
<h2>Extract the dump</h2>
<p>
Once downloaded, you will end up with a directory with another
directory within it named <em>Content</em>. Inside that
<em>Content</em> directory contains the data dumps of each site
compressed in a <a href="http://www.7-zip.org/">7-zip</a> file.
</p>
<p>
For each of the sites you wish to import into Stackdump, extract
the compressed file to a temporary location (each compressed
file contains another directory inside it with the actual data,
so you can just extract each file into the same temporary
location).
</p>
<h2>Import them into Stackdump</h2>
<p>
This process can take upwards of 10 hours or more depending on
the size of the dump you're trying to import.
</p>
<p>
Before you can import data though, you need to download the
required metadata so Stackdump can load the dumps properly.
</p>
<ol>
<li>Fire up a terminal/command prompt and navigate to the directory you extracted Stackdump into.</li>
<li>
Execute the following command -
<pre>./start_python.sh python/src/stackdump/dataproc/get_sites_info.py</pre>
</li>
</ol>
<p>
Now that you have the site metadata, you can import the dumps.
For each dump you wish to import, do the following -
</p>
<ol>
<li>Fire up a terminal/command prompt and navigate to the directory you extracted Stackdump into.</li>
<li>Find the directory containing the data dump XML files. This is likely to be a directory inside the temporary location you extracted to earlier. The directory will contain files like <em>posts.xml</em>, <em>users.xml</em> and <em>comments.xml</em>.</li>
<li>
Execute the following command, replacing <em>path_to_dir_with_xml</em> with the path from the previous step -
<pre>./start_python.sh python/src/stackdump/dataproc/import.py path_to_dir_with_xml</pre>
</li>
</ol>

55
python/src/stackdump/templates/nodata.html

@ -15,60 +15,7 @@ Stackdump - Import data instructions
data dump, then run some scripts on the Stackdump server. data dump, then run some scripts on the Stackdump server.
</p> </p>
<h2>Get the StackExchange data dump</h2> {% include 'includes/import_data.html.inc' %}
<p>
StackExchange data dumps are distributed using BitTorrent. You
will need a BitTorrent client like
<a href="http://www.utorrent.com">uTorrent</a> to download it.
</p>
<p>
The data dumps can be downloaded from
<a href="http://www.clearbits.net/creators/146-stack-exchange-data-dump">http://www.clearbits.net/creators/146-stack-exchange-data-dump</a>.
</p>
<h2>Extract the dump</h2>
<p>
Once downloaded, you will end up with a directory with another
directory within it named <em>Content</em>. Inside that
<em>Content</em> directory contains the data dumps of each site
compressed in a <a href="http://www.7-zip.org/">7-zip</a> file.
</p>
<p>
For each of the sites you wish to import into Stackdump, extract
the compressed file to a temporary location (each compressed
file contains another directory inside it with the actual data,
so you can just extract each file into the same temporary
location).
</p>
<h2>Import them into Stackdump</h2>
<p>
This process can take upwards of 10 hours or more depending on
the size of the dump you're trying to import.
</p>
<p>
Before you can import data though, you need to download the
required metadata so Stackdump can load the dumps properly.
</p>
<ol>
<li>Fire up a terminal/command prompt and navigate to the directory you extracted Stackdump into.</li>
<li>
Execute the following command -
<pre>./start_python.sh python/src/stackdump/dataproc/get_sites_info.py</pre>
</li>
</ol>
<p>
Now that you have the site metadata, you can import the dumps.
For each dump you wish to import, do the following -
</p>
<ol>
<li>Fire up a terminal/command prompt and navigate to the directory you extracted Stackdump into.</li>
<li>Find the directory containing the data dump XML files. This is likely to be a directory inside the temporary location you extracted to earlier. The directory will contain files like <em>posts.xml</em>, <em>users.xml</em> and <em>comments.xml</em>.</li>
<li>
Execute the following command, replacing <em>path_to_dir_with_xml</em> with the path from the previous step -
<pre>./start_python.sh python/src/stackdump/dataproc/import.py path_to_dir_with_xml</pre>
</li>
</ol>
</div> </div>
</div> </div>