Ceph - Troubleshooting

OSD

Display OSD with used > 80%:

ceph osd df | awk '/^ID/ || $17 ~ /^8/'

Display number of replicas by Pool:

ceph osd dump | awk '/pool/ {printf "%s;%s;%s\n", $2, $3, $6}' | column -s';' -t

Change weight of OSD:

ceph osd reweight 41 .9 # Set to 0,9
ceph osd reweight 41 1  # Set to 1

PG

Get PG with large omap objects:

pool=cephfs_metadata
pg_ids=$(ceph pg ls-by-pool ${pool} | awk '/^[0-9]+/ {print $1}')

for pg_id in ${pg_ids[@]}; do
    num_large_omap_objects=$(ceph pg $pg_id query | awk '/num_large_omap_objects/ {print $2}' | head -1 | tr -d ',')
    echo "$pg_id -> num_large_omap_objects: ${num_large_omap_objects}"
done

Crash

List of all crash:

ceph crash ls

Detail of a crash:

ceph crash info {id}
# Ex: ceph crash info 2023-06-20T08:41:00.895698Z_612f8cd6-14b2-4784-ab57-5b8d602d1259

Archive a crash:

ceph crash archive {id}
# Ex: ceph crash archive 2023-06-20T08:41:00.895698Z_612f8cd6-14b2-4784-ab57-5b8d602d1259

Archive all crashs:

ceph crash archive-all