this post was submitted on 24 Aug 2024
39 points (95.3% liked)

Linux

47287 readers
1676 users here now

From Wikipedia, the free encyclopedia

Linux is a family of open source Unix-like operating systems based on the Linux kernel, an operating system kernel first released on September 17, 1991 by Linus Torvalds. Linux is typically packaged in a Linux distribution (or distro for short).

Distributions include the Linux kernel and supporting system software and libraries, many of which are provided by the GNU Project. Many Linux distributions use the word "Linux" in their name, but the Free Software Foundation uses the name GNU/Linux to emphasize the importance of GNU software, causing some controversy.

Rules

Related Communities

Community icon by Alpár-Etele Méder, licensed under CC BY 3.0

founded 5 years ago
MODERATORS
AgreeableLandscape@lemmy.ml
nooter692@lemmy.ml
MarcellusDrum@lemmy.ml
cypherpunks@lemmy.ml
cyclohexane@lemmy.ml
d3Xt3r@lemmy.nz
39
How can I organize my ebook collection without file duplication? (programming.dev)
submitted 3 weeks ago* (last edited 3 weeks ago) by CoderSupreme@programming.dev to c/linux@lemmy.ml
23 comments fedilink hide all child comments
 

I am seeking advice regarding my ebook collection on a Linux system, which is stored on an external drive and sorted into categories. However, there are still many unsorted ebooks. I have tried using Calibre for organization, but it creates duplicate files during import on my main drive where I don't want to keep any media. I would like to:

  • Use Calibre's automatic organization (tags, etc.) without duplicating files
  • Maintain my existing folder structure while using Calibre
  • Automatically sort the remaining ebooks into my existing categories/folder structure

I am considering the use of symlinks to maintain the existing folder structure if there is a simple way to automate the process due to my very large collection.

Regarding automatic sorting by category, I am looking for a solution that doesn't require manual organization or a significant time investment. I'm wondering if there's a way to extract metadata based on file hashes or any other method that doesn't involve manual work. Most of the files should have title and author metadata, but some won't.

Has anyone encountered a similar problem and found a solution? I would appreciate any suggestions for tools, scripts, or workflows that might help. Thank you in advance for any advice!

all 24 comments
sorted by: hot top controversial new old
[–] solrize@lemmy.world 10 points 3 weeks ago (1 children)

If the files are literally duplicated (exact same bytes in the files, so matching md5sums) then maybe you could just delete the duplicates and maybe replace them with links.

Automatically sorting books by category isn't so easy. Is the metadata any good? Are there categories already? ISBN's? Even titles and authors? It starts to be kind of a project but you could possibly import MARC records (library metadata) which have some of thatinfo in them, if you can match up the books to library records. I expect that the openlibrary.org API still works but I haven't used it in ages.

[–] silkroadtraveler@lemmy.today 9 points 3 weeks ago

Migrate to Calibre and use Calibre Virtual Libraries. However based on the comments I’m reading, it looks like you want something that is not application based. Good luck with that.

[–] rand_alpha19@moist.catsweat.com 3 points 3 weeks ago

  1. Open the Preferences in Calibre

  2. Click on "Saving books to Disk" (found under Import/Export)

  3. Make sure "Save cover separately," "Update metadata in saved copies," and "Save metadata in separate OPF file" are all unchecked.

  4. Adjust the "Save template" to the filename format that you prefer. You can use variables as folder names so, for example, {author_sort}/{title} would put everything by Stephen King in a folder titled "King, Stephen" and each book would be inside of a self-titled folder.

  5. Select all of the books you want, then click the floppy disk icon and save them to a temporary directory.

  6. Delete the old library, then import a new library (with the new filenames) from the temporary directory.

  7. Delete the temporary directory.

Or you can just use symlinks. :P

[–] paddirn@lemmy.world 3 points 3 weeks ago* (last edited 3 weeks ago) (2 children)

I’ve run into exactly the same issue with my large ttrpg ebook/pdf collection (+100k file data hoarding… it’s not a problem, I swear) and I’ve not really found a good option I’m entirely happy with. Calibre duplicates everything and I don’t like the thought of having my collection’s organization tied to a specific piece of software if I just delete my duplicates. Plus I’m elitist and think the UI/logo are gross to look at.

Zotero is the least worst option I’ve found, but it’s geared towards scholarly journals and such, so not great, but serviceable. Not sure if it’s on linux though.

Jellyfin is apparently able to handle ebooks with a plugin, though I didn’t particularly care for it when I tried it months ago.

There’s a handful of other ebook software out there, mostly geared towards comics/manga, so depending on what you have those might be worth looking for.

I’d like to use Obsidian for it and just turn the directory into a vault and let it automatically scan the folders for files, but that doesn’t work great either.

The best piece of software I’ve seen that could potentially handle it is an app called Stashapp… which is unfortunately geared towards adult film. But it’s feature-set if it could be applied to PDFs seems like it would be ideal.

[–] astro_ray@piefed.social 2 points 3 weeks ago

Zotero is on Linux, and it has a LibreOffice plugin as well. Though, I do not like Zotero. Zotero is more geared towards reference management, but it also offers some pdf, epub management. But I find their document management too tedious. It's just easier for me to just rename files. That served me well for a long time.

[–] conciselyverbose@sh.itjust.works 1 points 3 weeks ago* (last edited 3 weeks ago) (1 children)

Yeah, I've tried, both for actual files and for tracking my reading across multiple platforms, and nothing really seems to fit my needs, especially when I want to actually read them on an Android ereader. Anything I choose seems to have a lot of manual effort, frequently, or just a dumpster fire of an actual reading experience.

I feel like I'm eventually going to have to make my own, which is fine, I guess, but I'm definitely not comfortable actually managing a community project or just building up the codebase or documentation to the level someone else would be enthusiastic to use as a jumping off point to manage themselves, so it will probably just stay a personal project that ends up not helping anyone else solve the same problems I have.

[–] rand_alpha19@moist.catsweat.com 2 points 3 weeks ago (1 children)

Have you tried Kavita? I use it to read comics and e-books on my Android tablet and my Kindle Paperwhite. It also uses OPDS so it has compatibility with some reading apps too, like KOReader, FB Reader, Mihon/Tachiyomi, Moon+ Reader, etc.

Website

Demo - Username: demouser Password: Demouser64

[–] conciselyverbose@sh.itjust.works 1 points 3 weeks ago* (last edited 3 weeks ago) (1 children)

I'm aware of it and explored it a little, but the folder structure requirements are the opposite of what I'm interested in. I want to dump everything in one place and use the UX of my reader to manually build series, adjust metadata, and do everything else.

Most of the benefits of it are really only useful in its browser based reader, which is also a dealbreaker, and it doesn't really add anything to Moon Reader because OPDS integration doesn't actually sync anything, which is the whole reason I'd want a dedicated server over just having everything in a cloud drive.

It's cool if it works for you, but it doesn't really solve any of the problems I want solved.

[–] rand_alpha19@moist.catsweat.com 2 points 2 weeks ago (1 children)

Hm, well, hopefully my other comment helps you then. I don't think there's an automated tool for this — though a shell script might do the trick, or at least get you most of the way, if you have basic scripting knowledge.

[–] conciselyverbose@sh.itjust.works 1 points 2 weeks ago* (last edited 2 weeks ago) (1 children)

I don't want anything automated. I just want to be able to do it manually with a database that handles all of the metadata and organization and literally no folders but the top level one containing every file. Calibre's insistence on me either having incorrect author information or splitting everything with multiple authors into unique folders for every combination is most of the reason I can't stand it. The actual bulk editing tools are good. The end result of a mess of folders isn't.

I'm not OK with folders, especially nested folders.

[–] rand_alpha19@moist.catsweat.com 1 points 2 weeks ago (1 children)

Okay? My other comment might help you then, you can change in the preferences whether to put your library in nested folders or not.

If all else fails, make a post on the MobileRead forums; there are lots of nice and knowledgeable book people there with tons of Calibre experience.

I'm not trying to get you to do something you don't want to, so your wall of text doesn't really make sense to be directed at me. I didn't make Calibre.

[–] conciselyverbose@sh.itjust.works 1 points 2 weeks ago

Wall of text?

I answered a suggestion with very clear preferences that something not use a trainwreck of a directory structure and you pointed me to a post on how to make a trainwreck of a directory structure, so I wrote a short paragraph again clarifying that I'm not OK with that.

[–] linearchaos@lemmy.world 3 points 3 weeks ago (1 children)

I tried to ingest a four terabyte epub library once. Even getting the data ingested with the author and title in the right spot was almost impossible. If a duplicates weren't just slightly wrong would be a different story but the duplicates are often misspells or different spellings.

Realistically the best thing you can do is get an output of file name, title, author and hand dedupe, but even then you're going to have to be careful about quality and language and all kinds of other strange issues you run into with large libraries.

In the end I gave up and only stored what I really wanted and would realistically ever need and that was small enough to hand cull.

[–] beejjorgensen@lemmy.sdf.org 2 points 3 weeks ago (1 children)

When I had to match against misspells I found Levenshtein distance to be most useful.

https://en.wikipedia.org/wiki/Levenshtein_distance?wprov=sfla1

[–] linearchaos@lemmy.world 2 points 3 weeks ago

Yeah, I wrote some python once to give me voice control over a plex server. Distance algorithms did okay I had a lot better results out of fuzzy wuzzy though

Kind of a dicey prospect with risk when you're doing duplication though.

Then you run into problems with things like Harry Potter and the philosopher's Stone versus Harry Potter and the sorcerer's Stone. Depending on how badly your database is degenerated you can even end up with words out of order and all kinds of crazy crap. If the database is truly large just truing the database up can be unreasonably time-consuming.

I was pretty amazed at all the different versions of string search and comparison algorithms.

[–] yesman@lemmy.world 3 points 3 weeks ago (1 children)

Why is duplicate files a problem?

[–] CoderSupreme@programming.dev 1 points 3 weeks ago* (last edited 3 weeks ago) (1 children)

I don't like keeping duplicate files, especially in my main drive where I don't store media. If I didn't mind duplicate files, it wouldn't be an issue.

[–] Ledivin@lemmy.world 6 points 3 weeks ago* (last edited 3 weeks ago)

I guess I'm a little confused... why are there duplicates at all? What operation are you performing that ends with duplicates?

Is this drive just where you download them to, and then move them to your organized drive? Why do the books ever touch this drive at all? If there isn't supposed to be media on the drive, why not just delete the source folder after the organization task is complete?

[–] j4k3@lemmy.world 2 points 3 weeks ago

Distilbert https://huggingface.co/distilbert/distilroberta-base

...was setup for something like that here but note that the repo that runs this has an "unsafe" warning that I have not looked into: https://huggingface.co/spaces/nasrin2023ripa/multilabel-book-genre-classifier

https://huggingface.co/spaces/nasrin2023ripa/multilabel-book-genre-classifier/tree/main

It might be fine or whatnot, I'm on mobile and can't see the file in question. The associated Python code might be a helpful starting point.

In my experience, most models intentionally obfuscate copyright sources. They all know the materials to various degrees, but they are not intended to replicate sources. They all have strong interference in place to obscure both their recognition and reproduction potential. If, for instance, you can identify where errors are inserted and make a few corrections, they often continue adding a few details that are from the original source. If this is done a few times in a row, they tend to gain more freedom before reverting to obfuscation again. This is the behavior I look out for. It is a strong tool too if you get creative in application.

Perhaps someone posts an API to look up the library of congress classification of a work based on a few lines or something. GL

[–] thejoker954@lemmy.world 0 points 3 weeks ago

In regards to another reply, there are multiple options of dealing with duplicates in calibre. From merging to deleting them.

It's been a while since I had to do any library setup with calibre and Im kinda confused by what you are saying with "but it duplicates files when importing in a different drive where I don’t want ebooks"

Do you have multiple libraries in calibre?

Are you using multiple library managers?

Why not just use calibres folder structure?

And also if you don't know about it - mobileread forums has been around a long time and has a whole section on calibre where you could probably get more specific help.