We all know the “school of magic for gifted teenage wizards” book trope has been pretty well covered by this point. However, even if you're sick of the very concept, I still recommend you give Naomi Novik's Scholomance trilogy a shot. (If you enjoyed her previous novel Uprooted, you'll especially have nothing to fear about the quality.)

Here comes the most-obvious-comparison-ever elevator pitch:

Imagine if Hogwarts were not only a dangerous place to send your kid, but almost certainly to be lethal for them, and that they stand close to zero chance of surviving it, much less graduating. Imagine that the school lacked any teachers whatsoever to guard and steward the students, instead assigning tasks and schedules to them of its own semi-sentient will. Imagine that, despite its dangers, the school is actually preferable to the outside world, which is infested with magic-eating monsters.

Now imagine a protagonist who, rather than being destined to defeat the “Dark Lord” of this world, is in fact a latently powerful sorceress prophesized to be the Dark Lord.

Here's the description from book one, A Deadly Education:

Lesson One of the Scholomance: Learning has never been this deadly.

A Deadly Education is set at Scholomance, a school for the magically gifted where failure means certain death (for real) — until one girl, El, begins to unlock its many secrets. 

There are no teachers, no holidays, and no friendships, save strategic ones. Survival is more important than any letter grade, for the school won’t allow its students to leave until they graduate… or die! The rules are deceptively simple: Don’t walk the halls alone. And beware of the monsters who lurk everywhere.

El is uniquely prepared for the school’s dangers. She may be without allies, but she possesses a dark power strong enough to level mountains and wipe out millions. It would be easy enough for El to defeat the monsters that prowl the school. The problem? Her powerful dark magic might also kill all the other students.

This review post by Dan of One Man Book Club gives you an idea of what to expect from the series' setting:

Inside The Scholomance it’s The Hunger Games x The Lord of the Flies x H.P. Lovecraft. The food might be poisoned, one wall of your room opens to an endless void, and there are floor drains for ease in cleaning up after all the spilled guts.

Needless to say, this is not a series for kids.

Here are the names of the three books:

1. A Deadly Education
2. The Last Graduate
3. The Golden Enclaves

Once you've read the series, I suggest listening to this episode of The Incomparable to enjoy some spoiler-filled discussion by a group of smart readers like yourself ;)