Note the code I present in this answer is not intended to be a solution, but just to explore some ideas.

initial idea

The first thing that came to mind was to convert each RGB color to HSV (Hue, Saturation, Value) representation. In this representation, the "Hue" component tells you what color it is (that value is a real number between 0 and 360 that represents an angle, which is the orientation of an imaginary vector in color space). The "S" component is the "Saturation" which tells you how much white that color has added to it. It is an integer between 0 and 1, where 0 is the minimum saturation (white) and 1 is the maximum (no white, pure color). Finally, the V is the "value" of luminosity that comes to say how much black is mixed with the color (0 is all black, 1 is no black at all, "pure" color).

I thought that by transforming each color in this way and simply ordering the resulting trios, they would already be grouped by "chromatic similarity", since we would be ordering first by Hue (color) and in the case of the same color, by saturation and then by brightness.

Unfortunately the results are not what I expected. The following python code is used to generate a bunch of random RGB hex values:

import random

def color_aleatorio():
  r, g, b = random.randint(0,255), random.randint(0,255), random.randint(0,255)
  return "#{:02x}{:02x}{:02x}".format(r,g,b)

mil_colores = [color_aleatorio() for _ in range(1000)]

Some elements of the resulting list:

['#6264c0',
 '#b37f4d',
 '#116dcc',
 '#a8ce4b',
 '#7ea6f8',
 '#a94f89',
 '#83be23',
 '#6c8052',
 '#d87e04',
 '#d107f2'
 ...]

The graphic representation of this "palette" of colors, in the same random order in which they were generated, would be as follows:

I got this representation with this code, in case someone wants to replicate it:

import matplotlib.pyplot as plt
import numpy as np
import matplotlib.colors

def mostrar_colores(colores):
  data = matplotlib.colors.to_rgba_array(colores)
  plt.imshow(np.array(data).reshape((20,50,4)))
  plt.grid(False)

mostrar_colores(mil_colores)

The following function implements the conversion of one of those values ​​to HSV form:

import matplotlib

def rgb2hsv(hex_rgb):
    r, g, b, a = matplotlib.colors.to_rgba(hex_rgb)
    r, g, b = r/255.0, g/255.0, b/255.0
    mx = max(r, g, b)
    mn = min(r, g, b)
    df = mx-mn
    if mx == mn:
        h = 0
    elif mx == r:
        h = (60 * ((g-b)/df) + 360) % 360
    elif mx == g:
        h = (60 * ((b-r)/df) + 120) % 360
    elif mx == b:
        h = (60 * ((r-g)/df) + 240) % 360
    if mx == 0:
        s = 0
    else:
        s = df/mx
    v = mx
    return h, s, v

Now we can ask Python to sort the sequence mil_coloresusing the above function as the sort key:

colores_ordenados = sorted(mil_colores, key=rgb2hsv)

The first ten elements already ordered would be these:

['#d4d4d4',
 '#efc7c7',
 '#df2b2a',
 '#c1201f',
 '#a30907',
 '#f09b99',
 '#9a2623',
 '#efcecd',
 '#753836',
 '#e30b01',
 ...

Which graphically gives this slightly disappointing result:

We can see that, yes... more or less they have been grouped by color, being able to appreciate the "rainbow strip" with orange above, violet below... but within each strip the order seems a bit random and is due to that values ​​with a very close HUE, but different values ​​or saturations end up very close.

second thought

Then I realized that what we want is for nearby colors to end up together. This forces us to define the concept of distance between colors.

A color at the end is nothing more than a point in a space of three coordinates: R, G, B. Therefore we can use the Euclidean distance between them (root of the sum of squares of distances between each coordinate).

Although the distance is now well defined, achieving an arrangement in which close colors end up together is basically another form of the traveling salesman problem . That is, we must find a path of all points that minimizes the distance traveled. That tour would be the desired ordering of colors.

Since that problem is NP-hard, I've given up on implementing it

Update. Reflections and heuristics

After some more thought, I come to the conclusion that sorting "by distance" is really just one more way to sort them, and not necessarily the best one, since the concept of "best" is not well defined here. In the "proximity" sort, colors can appear separately that in another sort would be considered to go together, for example, a very dark red could appear "with the blacks" instead of "with the reds". Which is better?

I think the problem at the bottom cannot be solved because it is not well defined. The input color set can actually be understood mathematically as a set of points in 3-dimensional space (which would be its components, either RGB or HSV). By asking for an ordering, basically we want to pass that to a single dimension. Therefore it is a projection problem .

It is somewhat equivalent to the problem of mapping the surface of the earth's sphere on a plane. There is no single solution, and therefore there are different cartographic projections, depending on which aspect we give priority to. Some projections are useful because they make navigation easier by keeping the angles constant. Others keep the areas of the countries constant and are more useful to get an idea of ​​their relative size, etc.

Something similar happens with color ordinations. If all the colors were "pure" (in HSV they would have S=1 and V=1) their ordering would be simple, based only on their "Hue", and it would be like this:

But if we also have "pastel" colors (in which the saturation is lowered and therefore they are closer to white), or "dark" colors (in which the brightness is lowered and therefore they are closer to black), there is nowhere to color them. in the impeccable hue arrangement just seen.

This paintbox shows how the manufacturer has solved the problem:

He has made two drawers. The top one for saturated colors (sorted by hue) and the bottom one for pastel tones (also sorted by hue). The "dark" colors have been interspersed according to their hue. For example a dark blue at the end of the blues. That "breaks" the beautiful rainbow because after that dark blue comes a lighter green-blue. Furthermore, he has only considered two levels of saturation (the two drawers). There could be many more...

Is this arrangement better or worse than another? Again it depends on what is intended. The artist may want to have the paints with the same hue together and care little whether or not the "rgb distance" between the colors matches the distance between paints in his box.

In short, there is no defined criterion for preferring one arrangement over another.

All that said, I present another heuristic to order the thousand random colors that is based on:

1. Separate the very dark colors and leave them for the end
2. Separate the remaining colors by their saturation, putting the most saturated ones first and then the "pastel tones".
3. Sort each sequence by its hue, but reducing the number of possible hue from infinity (since the hue is a real that varies continuously between 0 and 360) to only 15 categories (an arbitrary value based somewhat on the easily discernible hue ), and equal to hue, for its brightness.

This is the algorithm (python):

def ordenacion(hex_rgb):
  h,s,v = rgb2hsv(hex_rgb)
  return  v<0.002, s<0.7, int(h/360*15), v

colores_ordenados = sorted(mil_colores, key=ordenacion)

And this is the result:

If instead of a thousand colors I use ten thousand, the result is more impressive, going from this:

to this:

To show that the algorithm also scales for a few colors, these are the ones proposed by Alvaro:

c = [ "#fff000", "#238923", "#aaaaa0", "#ff2300", "#2ff014", "#010203" ]

And this is the ordering that results:

['#ff2300', '#fff000', '#2ff014', '#238923', '#aaaaa0', '#010203']

Your problem is simpler to solve than it seems at first glance if you understand a couple of things first.

1. Sorting colors is an impossible task but you are not trying to sort all colors, just a small collection.
2. La parte difícil de ordenar cualquier colección es determinar cual de entre dos elementos tiene mas precedencia que el otro. Precedencia significa que va primero, no necesariamente que su valor sea menor pues en una lista ordenada descendente los mayores valores van primero.

Los algoritmos de ordenamiento para esto son ampliamente conocidos y puedes escoger entre un gran número de ellos con ventajas y desventajas entre ellos. En todos los casos es necesario poder determinar cuando un elemento tiene más precedencia que otro. En esa parte es en la que debes concentrarte.

Para comparar debes saber una cosa primero y es que los colores no se pueden representar usando un "círculo de color" como pretendes pues dependiendo del formato a usar (en tu caso RGB) siempre son una figura geométrica tridimensional. En el caso de RGB se usa un cubo como este:

By SharkD - Own work. 
Download source code., CC BY-SA 4.0, https://commons.wikimedia.org/w/index.php?curid=70991647

Es posible usar un cilindro para representarlo teniendo en cuenta un par de cosas:

Que tan rojo azul o verde es un color depende del empuje del componente o los componentes de color más fuerte hacia su base (el más puro), o sea un vector que parte del eje transversal del cilindro hacia su borde.

Por ejemplo el rojo puro FF0000 es exactamente igual a este otro rojo más oscuro #BE0000. Esto es difícil de digerir a simple vista pero si lo representas con una imagen te darás cuenta que la única diferencia es que uno es más brillante que otro, o sea que el rojo puro se encuentra más arriba en el eje vertical del cilindro de colores.

Rojo puro

Rojo oscuro

Si te fijas bien te darás cuenta que esta representación coincide más menos con el modelo HSL donde el color se expresa con grados formando un círculo. En realidad ni HSL ni HSV son cilindros tampoco sino figuras formadas por conos pero el hecho que puedan representarse de forma circular es suficiente para tus propósitos.

En ambos modelos se puede deducir un par de cosas:

1. El blanco #FFFFFF, el negro #000000 y los puros grises (igual valor de R, G y B) son el mismo color variando por el brillo pero que no tienen afinidad por ningún componente en particular lo que cambia es su posición en el eje vertical de acuerdo al modelo escogido.

2. Que existen colores que a simple vista parecen negros pero en realidad son rojos, verdes, etc, pero que son muy oscuros o claros para decirlo de una manera muy sobre-simplificada.

3. Que no puedes decidir por un sólo parámetro, el color, sino que debes tener en cuenta otros como luminosidad, chroma o valor de acuerdo al modelo escogido. El resultado final será diferente de acuerdo a tus preferencias.

Las prioridad de colores impuesta por tu problema requiere que el violeta puro (#ff00ff) se encuentre el primero en la lista y este se encuentra exactamente a los 300^o grados de la circunferencia de color así que lo único que debes hacer para obtener los valores que deseas es rotar 60 grados la circunferencia en sentido contrario a las manecillas del reloj.

Aquí te dejo el algoritmo:

// girar 60 grados el valor de hue
function offsetHgrad(color) {
  const offset = 60;
  const start = 360 - offset;
  const h = color.h >= start ? color.h - start : color.h + offset;

  return {
    h,
    s: color.s,
    l: color.l,
  }
}

// convertir rgb en hsl, algoritmo genérico
function rgbToHsl(color) {
  const rbgReg = /#([0-9a-f]{2})([0-9a-f]{2})([0-9a-f]{2})/;
  const [, red, green, blue] = rbgReg.exec(color);
  const r = parseInt(red, 16) / 255,
    g = parseInt(green, 16) / 255,
    b = parseInt(blue, 16) / 255;
  let max = Math.max(r, g, b),
    min = Math.min(r, g, b);
  let h, s, l = (max + min) / 2;

  if (max === min) {
    h = s = 0; // achromatic
  } else {
    let d = max - min;
    s = l > 0.5 ? d / (2 - max - min) : d / (max + min);
    switch (max) {
      case r:
        h = (g - b) / d + (g < b ? 6 : 0);
        break;
      case g:
        h = (b - r) / d + 2;
        break;
      case b:
        h = (r - g) / d + 4;
        break;
    }
    h /= 6;
  }

  return {
    h: Math.floor(h * 360),
    s: Math.floor(s * 100),
    l: Math.floor(l * 100)
  };
}

// función de comparación para algoritmo del mismo tipo
function compare(color1, color2) {
  const values1 = offsetHgrad(rgbToHsl(color1));
  const values2 = offsetHgrad(rgbToHsl(color2));

  if (values1.h !== values2.h) {
    return values1.h - values2.h;
  }
  if (values1.s !== values2.s) {
    return values1.s - values2.s;
  }
  return values1.l - values2.l;
}

document.addEventListener('DOMContentLoaded', () => {
  const sample = document.getElementById('sample');
  const expected = document.getElementById('expected');
  const result = document.getElementById('result');

  function addToDoc(container, color) {
    const box = document.createElement('div');
    box.className = 'box';
    box.style.backgroundColor = color;
    box.innerText = color;
    container.appendChild(box);
  }

  const input = ["#fff000", "#238923", "#aaaaa0", "#ff2300", "#2ff014", "#010203"];
  const intended = ["#ff2300", "#fff000", "#aaaaa0", "#2ff014", "#238923", "#010203"];
  input.forEach((i) => addToDoc(sample, i));
  intended.forEach((i) => addToDoc(expected, i));
  input.sort(compare).forEach((i) => addToDoc(result, i))
});

#sample,
#result,
#expected {
  display: flex;
  flex-wrap: nowrap;
}

.box {
  border: solid 1px black;
  margin: 5px;
  padding: 5px;
  color: white;
}

<div>Muestra</div>
<div id="sample">

</div>
<div>Resultado esperado</div>
<div id="expected">

</div>
<div>Resultado obtenido</div>
<div id="result">

</div>

Como ves la solución es bastante simple una vez que cambias el modelo mental y comparas usando los grados de una circunferencia en lugar de usar valores de color.

Si lo pruebas con los datos de @abulafia te das cuenta que hay violetas en ambos extremos pero que los menores son los violetas que tienen un mayor componente azul que es exactamente lo que pides.

function offsetHgrad(color) {
  const offset = 60;
  const start = 360 - offset;
  const h = color.h >= start ? color.h - start : color.h + offset;

  return {
    h,
    s: color.s,
    l: color.l,
  }
}

function rgbToHsl(color) {
  const rbgReg = /#([0-9a-f]{2})([0-9a-f]{2})([0-9a-f]{2})/;
  const [, red, green, blue] = rbgReg.exec(color);
  const r = parseInt(red, 16) / 255,
    g = parseInt(green, 16) / 255,
    b = parseInt(blue, 16) / 255;
  let max = Math.max(r, g, b),
    min = Math.min(r, g, b);
  let h, s, l = (max + min) / 2;

  if (max === min) {
    h = s = 0; // achromatic
  } else {
    let d = max - min;
    s = l > 0.5 ? d / (2 - max - min) : d / (max + min);
    switch (max) {
      case r:
        h = (g - b) / d + (g < b ? 6 : 0);
        break;
      case g:
        h = (b - r) / d + 2;
        break;
      case b:
        h = (r - g) / d + 4;
        break;
    }
    h /= 6;
  }

  return {
    h: Math.floor(h * 360),
    s: Math.floor(s * 100),
    l: Math.floor(l * 100)
  };
}

function compare(color1, color2) {
  const values1 = offsetHgrad(rgbToHsl(color1));
  const values2 = offsetHgrad(rgbToHsl(color2));

  if (values1.h !== values2.h) {
    return values1.h - values2.h;
  }
  if (values1.s !== values2.s) {
    return values1.s - values2.s;
  }
  return values1.l - values2.l;
}


document.addEventListener('DOMContentLoaded', () => {
  const sample = document.getElementById('sample');
  const expected = document.getElementById('expected');
  const result = document.getElementById('result');

  function addToDoc(container, color) {
    const box = document.createElement('div');
    box.className = 'box';
    box.style.backgroundColor = color;
    box.innerText = color;
    container.appendChild(box);
  }

  const input = ['#6264c0',
    '#b37f4d',
    '#116dcc',
    '#a8ce4b',
    '#7ea6f8',
    '#a94f89',
    '#83be23',
    '#6c8052',
    '#d87e04',
    '#d107f2'
  ];
  input.forEach((i) => addToDoc(sample, i));
  input.sort(compare).forEach((i) => addToDoc(result, i))
});

#sample,
#result {
  display: flex;
  flex-wrap: nowrap;
}

.box {
  border: solid 1px black;
  margin: 5px;
  padding: 5px;
  color: white;
}

<div>Muestra</div>
<div id="sample">

</div>
<div>Resultado obtenido</div>
<div id="result">

</div>

Puedes mejorar los algoritmos, el punto que me ocupa es que entiendas cuales son las partes importantes. Yo use el sort de javascript que no tiene un algoritmo definido pero al menos en Chrome usará una variante de Merge Sort llamada TimSort. En otros navegadores puede que usen Insertion sort, el resultado es realmente variable.

Ideally, you would be able to convert all three HSL values ​​to a single integer or string and use an integer algorithm like radix sort that can be faster than a comparison algorithm. Note that this requires an additional conversion step for each HSL value, so you need to assess whether there are actually benefits to using it. For very large collections the benefit will be obvious. The reality is that the data set (best, average, worst case, its size, etc) affects the performance of each comparison algorithm so you must decide which one you want to use. There is no infallible answer but advantages and disadvantages such as speed, complexity and memory consumption.