Roadmap

Version 0.0 MVP (Done)

  • Roadmap
  • About page
  • Scrape SB and SV
    • Parse tags from titles
  • Regularly crawl SB and SV
  • Tag list page
  • Basic search (no suggestions)
    • Only search order by date

Version 0.1 Actually Useful (Done)

  • Tag autocomplete suggestions in search
  • Sort by options
  • Meta-rating
  • Actual Front page with some content
    • Currently trending fics

Version 0.2 User Collaboration (Done)

  • User Accounts
    • Registration, login, etc
    • Should we bother even having email recovery initially?
    • Captcha for creating accounts / after failed login attempts?
  • User tagging
  • Suggestion of tag merging
  • Adding of new tags by users
  • User 'likes'?

Version 0.3 Tag improvements (Done)

  • Tag groups
    • Character Insert
      • Self Insert
      • OC Insert
      • Friend Insert
      • etc
  • Advanced search
    • Title search
    • Author search
    • Summary search?
    • Exclude tags
    • Filters (ie wordcount > x)

Version 0.3.5 Crawler refactor and rename detection (Done)

  • Move crawler synonymns into database and make them editable in tags
  • Use synonyms for autocomplete suggestions (maybe)
  • Sort site specific IDs in the database in a searchable way
    • Allows story rename detection
  • Re-crawl stories occasionally to update rating, not just when there are new chapters

Version 0.4 Ao3 (Done)

  • Regularly crawl certain large fandoms in Ao3
  • Ao3 de-tagging
  • Safe Search
  • Assess performance implications
  • Assess if we hit any hosting bottlenecks
  • This will take us from indexing ~32,000 stories to > 7 Million

Version 0.4.1 Bug Fixes / Misc (Done)

  • Manual story deleting / merging for crossposts
    • Store author -> site mappings for future improved crosspost detection
  • Actually serve static files correctly (oops)
  • Filter by rating
  • Figure out how the spider keeps locking up the database
  • Fix tag deletion (causes spider crashes in its current form)
    • Delete some useless tags imported from AO3
  • Fix tag merging to not create duplicate tag references
    • If two tags both tag the same story and are merged, the story will now have two references for the same tag.
  • Fix the xenforo spider to store its tag synonyms separately from the AO3 synonyms, theres been some really weird cross polination going on and some weirdly inaccurate tags showing up
  • Fine tune the AO3 meta-rating, its currently seems inflated somewhat
  • Fix sanitized HTML that made it into Tag names and Story names
    • Mostly just issues with &'s AFAIK
  • Searching for tags containing / does not work. Its getting unencoded early and messing with the route 404ing.
  • Tag autocomplete improvements
    • Hitting enter should select the first item
    • Order tags also by fandom -> freeform -> relationship -> character
    • Using / adds in wildcards to the search, FirstName/Firstname searches pretty good for relationships
  • Switch + button on tags in results into a - button that will exclude tags from the results

Version 0.5 Fanfiction.net (Done)

  • Regularly crawl parts of fanfiction.net

Version 0.6 More bug fixes and improvements

  • Add a filtering search for tags instead of a paginated page for each type of tag. Its impossible to find certain tags right now.
  • Clicking on a tag to go to it in the search box does not work on mobile
  • Follow stories and see updates on one page for all sites
    • Basic follows page
    • Marking stories as read
    • Shelve stories that you want to keep track of but aren't currently reading
    • Work for logged in and logged out users
  • Cloudflare has an approved bots things, but likely too small for that for now
  • Log searched fandoms and ensure we crawl heavily searched fandoms frequently
  • Upgrade hosting for more RAM (currently limited to 100MB) - DONE!!!!
    • If the site is slow to respond, its probably using too much RAM and restarting under the hood

Version 0.7 Server upgrade + include all of FFN and AO3

  • Begin to crawl all of Ao3 (not just updated stories)
  • Begin to crawl all of FFN (1 page every 40ish minutes...)
  • Customized front page

Version 0.71 Bug fixes

  • Fix issues with parsing authors on FFN pages (done)
    • Update crawler to auto fix incorrect authors from previous bug (todo)
  • Fix issues using tags with certain characters

Version 0.8 Non-fanfiction (some just call that fiction...) (in progress)

  • Inplace tag editing
  • Only render some tags with a more button
  • Mark duplicate button
    • And duplicate aggregation admin page
  • Mark deleted button / removal of dead links
  • Automate crawling of fandoms/tags that are more commonly searched
  • Attempt to add crawling for other sites like
    • Royal Road
    • Scribblehub
    • etc

Version 0.9

  • Attempt to add crawling for other sites like
    • Fictionpress
    • QQ (might wait for XF2 upgrade)
    • etc (and any missed fanfiction/fiction sites)

Possible Pitstops

AI Tagging

  • Train model on existing tags and have it suggest tags for stories based on content
  • Useful for adding sites that don't have any tags
  • Display AI tags separately and add a filter for them

AI Chapter Summaries

  • Some fics have years in between updates. Would be nice to auto generate summaries as a refresher.

User front page

  • Users can subscribe to tags and see new/trending/etc ficts for those tags
  • Watching fics, see latest update of fic and ability to mark as read

Costs?

  • If hosting costs get too high / if we get kicked off low tier hosting
  • Patreon?
  • Sponsored stories?

Better desktop experience

  • Optimized for Mobile currently, looks not great on desktop

The Queue

  • A queue that users can go to to view stories that need tagging. Would prioritize newer/untagged stories first.
  • Gamification? Meh

Mutlilanguage support

  • Currently only English