I use Manjaro Linux with the Cinnamon desktop and sometimes run into system-level issues, but I have no idea how to properly debug them. It doesn’t feel as straightforward as debugging a normal program. What’s the best way or resource to learn system debugging on Linux?

  • DigitalDilemma@lemmy.mlEnglish
    11·
    25 days ago

    Sysadmin here, this is my usual flow for various distros

    1. as /u/FigMcLargeHuge mentions, recent logfiles in /var/log. Notably /var/log/messages (EL) and syslog (Debian) but anything that’s recent.

    2. journalctl - More and more things are moving to binary logging. If you know the process, then journalctl -u processname restricts to just that. also add a -f for tailing it for ongoing logs.

    3. dmesg -T - especially at system level, this captures any hardware/low level logs. (-T reports actual times, not just seconds since boot)

    4. Once you have some logs that you think are related, but don’t know WTF they actually mean, you have two options. The first is to google likely strings. This is… ineffective much of the time - accidental misinformation and outdated advice is increasingly common. The answer might be there, but it takes time and can be frustrating to weed out the cruft.

    The better way, (IMO, and people downvote me for saying this) is to use AI. Get a few lines of logs with the errors, check them for confidential information, and simply paste the suspect lines into chatgpt, gemini, claude, co-pilot, whatever. No need for context, it’ll figure that out. The LLM will, 4 times out of 5, identify the problem very quickly.

    Now, once it’s identified that, it will offer to fix it for you. This is where you’ve got to be on your toes as LLMs are really really quick to give bad advice at this level. But that first triage is nearly always worth doing and helps shape your own mind as to what’s going on. AI is still useful for fixing it, but do understand what it’s telling you to do.