What's a species? The general idea is straightforward; it's a distinct, unique group of individuals who can reproduce with each other and that share some traits that make them distinct from all other individuals and groups. We all have an intuition that there are distinct groups--giraffes are giraffes in a way nobody else is; similarly, despite their variety, tomato plants are generally instantly recognizable as such.
Trying to get more concrete, even experts will give you slightly to significantly different answers. Folks working with different groups of organisms (plants, fungi, animals, bacteria) often focus on different aspects. So a short answer isn't possible. This page goes over some of the underlying principles, and my thinking on the topic as a flower-focused guy.
The there there
Underlying the whole business are some universally accepted aspects arising from the workings of evolution. The basic piece is that often new species arise from a genetic innovation that creates a difference with consequences. A novel flower may cater slightly better to a different pollinator than its parents and cousins. A plant may gain an advantage in growing in a slightly different environment than its group has historically favored. A tweak in reproduction may render a plant more efficient in reproducing with others bearing the same tweak rather than the ancestral population.
For the purposes of this discussion, I'm going to overlook another common mechanism of species origins where two populations become isolated and the chance wanderings of accidental changes cause the two populations to accumulate differences over time. Things like the loss of the land bridge that allowed genetic exchange between North America and China/Japan account for why so many of our native wildflowers have sister groups in these these remote places; more classical examples include the slow raising up of a mountain or opening of an ocean... or the creation of an island far from any existing land mass (Hawaii or the Galapagos, anyone?).
What matters: traits in focus
The tried-and-true answer to the question "What is the biological goal of an organism/its genes?" is 'production of viable/competitive kids'. In this light, we can view a flower (and even more specifically, its pollen/stigma) as the purpose of the entire plant. The success of these tiny parts is the success or failure of the individual. It amuses me to think that as an evolutionary entity, I am a dismal failure as I have neither reproduced nor materially assisted the viability or reproductive success of my near relations 😸.
The logical consequence, then is to look at all aspects of the structure and behavior of a flower as very likely places for 'speciating differences' to arise. Any changes in a flower's location, color, shape are opportunities to change interactions with pollinators or change which pollinators interacted with.
The Bruce World View: where the bullet hits the bone here is that in determining whether I think two groups are worth treating as different species depends on my perceiving and understanding how differences in their traits can cause them to have differing advantages... I tend to think "different species" if the flowers are different in a way that leads to different pollination prospects (example: spotted vs. yellow Fairy Bells), and I'm very dubious about cases where the primary/only clear differences involve, for example, leaf shape tweaks (example: sharp-lobed vs. round-lobed hepatica). Features like different spacings between leaves on the stem or branching patterns or subtle differences in flower number tend to leave me cold. I can well believe that as two groups have become separated, reliable differences have occurred in these characters, but I have a hard time believing these are the CAUSE of separation, and so I dislike ID keys that make me consider these to distinguish species.
An interesting case of floral distinctiveness: Canada Wild Ginger (Asarum canadense)
I've written an entire post about this one--there's a relatively distinct flower structure offshoot of Canada Wild Ginger that's often called Hydra Ginger... and sometimes classified as a species, Asarum acuminatum. Alas for folks like me who are rooting for acuminatum, evidence suggests that at present, it's "just" a variant of a single, possibly important, trait that has not (yet?) lead to a stable, differentiated offshoot.
The elephant in the room: reproductive isolation
I've been cheerfully avoiding one of the rock-solid species separators. If two groups cannot produce viable, reproducing offspring, they're different species. They must each 'stick to their own branch', and they'll like continue to accumulate differences as a change occurring in one group cannot 'float over' to the other.
Of course, even this lovely, clear delineation comes with caveats. For plants in particular, the answer is often far from black-and-white. One can find groups of plants that, when crossed, consistently give rise to fruitful offspring 1% or 3.5% or 62% or 93% as efficiently as crossing to members of their own group. Short of drawing some wholly arbitrary cutoff for 'reproductive efficiency', we're left with muddy water. And it takes some patient experimentation to even determine this result.
However, the bedrock is that if reproductive success is consistently 0%, not many would try to wrangle the groups into the same species. Though even here edge cases occur--what if A mates with B at 'pretty good efficiency', as B does with C, as C does with D... but an attempted mating of A with D gets us to 0%?
'Separatedness' is a feature where I find goldenrods (Genus Solidago--48 species in the Blue Ridge) and asters (for example, Genus Symphyotrichum--40 species in the Blue Ridge) to be fascinating. These are both very large groups, and overall the form and function of their flowers seem to be pretty darn similar. Yet walking in Blue Ridge woods and meadows, we see readily distinguishable groups that appear to have relatively sharp delineations (intermediates between two groups not being commonly observed) growing intermingled or with only easy 'pollinator flight' distance separating them. Further, these flowers are 'generalists' (serving several-to-many species of pollinators) so it's not easy to argue that the difference keeping them different lies in flower action. There are certainly groups that prefer more or less sunshine, but it's clear that there isn't 'the ONE that does best in shade' or 'the ONE that is a sun-lover'. But the fly in the ointment is that both experts and lay observers come away with the impression of distinct, somewhat well-delineated 'species'.
So what's going on in these large, wildly successful groups? What's keeping them separate? To date, I have no illuminating thoughts to offer, and have not found keys or publications that give me insight. It's an ongoing quest; please e-mail me with any insights, studies, or conjectures you're aware of!
A fascination of mine: 'changes distance'
In the gingers example above (Canadian v. Hydra), it seems like the driving difference is a single trait, and quite possible a difference of a single genetic change. That's pretty small potatoes for something I'm hoping will become a unified, separable entity. Indeed, one of the things I like to consider is "how much meaningful genetic change separates one group from another?" The word 'meaningful' is critical here; all of us (ignoring identical twins) have thousands of genetic differences... but virtually all of these have no manifestation at all in terms of our appearance, structure or function.
In this light, something I think I'm seeing (and others are seeing) in the genus Trillium is really interesting. The Trillium 'separatedness' story is fascinating, and I hope to explore some aspects of it in a post. Briefly, there are many 'meaningful' differences within the group--decidedly different structures (pedicellate/stalked vs. sessile/right-out-of-the-stem trilliums), smells (some stink; some smell sweet), colors (reds, yellows, whites) and flower location (above or below the umbrella-like foliage).
Greatly complicating matters, many 'distinct' groups produce fertile offspring quite cheerfully. Some of these complications and confusions are acknowledged by classing groups as 'complexes', such as the 'Trillium erectum complex'--a group of 'species' having genetic similarity to and interbreeding with Trillium erectum.
What I find particularly striking about the T. erectum complex is hybrid swarms--offspring of a cross between different members of the complex that themselves show a wide range of variation. Again, there's a post in the works (at least in my mind's eye), but as an example, in a naturally occurring hybrid swarm, which I'm pretty sure involves erectum complex members Trillium vaseyi and T. erectum (possibly the white varient of the latter... and possibly not!), there are dozens of flower variants that appear to be stable 'programs'--see the image below. BRIEFLY put, this implies that there are a significant number of changes in a significant number of different genes... in other words, there are lots of 'meaningful changes' in one parent compared to the other.


Giant, descending flower of Trillium vaseyii (Left) and the daintier, above-the-foliage flower of Trillium erectum, right. vaseyii blooms later, and may smell better. There's a white variant of erectum... but I've not seen it near the hybrid swarm individuals pictured below








Hybrid offspring from a naturally occurring (T. vaseyii x T. erectum(?) cross. Interestingly, several of the hybrids I have seen that bloom below the foliage (like vaseyii) also shower later-season blooming as vaseyii does. There also appears to be a gradient in how far beneath the leaves hybrids bloom, again suggestive of multiple genetic differences.