I wouldn't recommend what you want to do. Why are you using a List<T> in the first place? If you can tell us precisely what characteristics the data-structure that you want to create should have, and how it should interface with the consuming API, we might be able to give you a proper solution to your problem.
But I will try to answer the question as asked.
Can I do this without copying, like somehow get a pointer to the array used internally by List?
Yes, although you would be relying on an undocumented implementation detail. As of NET 4.0, the backing array field is called _items.
Vertex[] vertices = (Vertex[]) typeof(List<Vertex>)
.GetField("_items", BindingFlags.NonPublic | BindingFlags.Instance)
.GetValue(VList);
Do note that this array will almost certainly have slack at the end (that's the whole point of List<T>), so array.Length on this array won't be all that useful. The API that consumes the array would need to be notified of the "real" length of the array through other means (by telling it what the list's real Count was).